N88 - 17215 


TUTORING ELECTRONIC TROUBLESHOOTING IN A 
SIMULATED MAINTENANCE WORK ENVIRONMENT 


Sherrie P. Gott, PhD 
Air Force Human Resources Laboratory 
Manpower and Personnel Division 
Brooks AFB TX 78235-5601 


ABSTRACT 

Expert performances on authentic technical 
problems such as electronic fault isolation are 
being captured in “real time" to provide the 
basis for a new generation of Air Force training 
systems. Experts (and novices) in dozens of 
maintenance jobs in electronic and 
electro-mechanical domains are being studied 
with a hybrid knowledge engineering-cognitive 
task analysis methodology- A primary goal is to 
establish what humans really need to know and 
how they use their knowledge when they problem 
solve in complex workcenters that are saturated 
with "smart" machines- The cornerstone of the 
method is an expert problem solving dyad- One 
expert poses a problem and simulates equipment 
responses to a second expert who attempts to 
isolate the fault conceived by the first 
expert- Engineering expert knowledge in this 
fashion situates skill in the actual problem 
context and thus highlights the conditional i zed 
character of expert knowledge- This is in 
contrast to representation techniques that yield 
decontextual ized (and perhaps nonessential) 
declarative knowledge through interrogation of a 
single expert- A series of intelligent tutoring 
systems— or intelligent maintenance 
simulators— is being developed based on expert 
and novice problem solving data of this type. 

The training systems rest on the same 
problem-based cornerstone- A graded series of 
authentic troubleshooting problems provides the 
curriculum, and adaptive instructional 
treatments foster active learning in trainees 
who engage in extensive fault isolation practice 
and thus in condi tionali zing what they know. A 
proof of concept training study involving human 
tutoring was conducted as a precursor to the 
computer tutors to assess this integrated, 
problem-based approach to task analysis and 
instruction. Statistical ly significant 
improvements in apprentice technicians' 
troubleshooting efficiency were achieved after 
approximately six hours of training. 

INTRODUCTION 

Both military and industrial work environments 
have grown steadily in complexity in recent 


decades as technologies, particularly 
electronics related, have advanced at staggering 
rates. Today's workers find themselves in 
contexts where interacting with complex machines 
is the rule. And yet, the nature of intelligent 
performance in such machine interactions is not 
well understood. In addition, beliefs that 
cognitive demands on humans have diminished with 
the proliferation of so-called smart machines 
have diverted attention away from the human 
capabilities that are important for a high-tech 
workforce- Yet, it now seems clear that for the 
foreseeable future, machine diagnostic 
capabilities have definite limits- These limits 
in turn place a premium on the human expertise 
that is needed to pick up where the machines 
leave off. For example, the hit rate for some 
built-in diagnostics for the BIB is only 65 
percent. Even with today's widely used 
maintenance aiding machines (many having expert 
system features), the ratio of maintenance hours 
to flying hours for the F-15 aircraft is 50:1 
(Atkinson & Hiatt, 1985). In more general terns 
it has been estimated that as much as 90 percent 
of the life-cycle cost of a defense hardware 
system is the cost of maintaining it. 

A large-scale research program is underway at 
the Air Force Human Resources Laboratory in 
direct response to this problem. The goals are 
to develop methods for representing human 
expertise on complex technical tasks so that 
training systems capable of meeting the demands 
of high-tech workcenters can be realized. 

THE ENGINEERING OF CONDITIONAL I ZED KNOWLEDGE 

The knowledge engineering approach in the Air 
Force Basic Job Skills (BJS) Research Program 
involves "real-time" problem solving, multiple 
stages and types of knowledge engineering 
inquiry, and a number of formats for knowledge 
representation- A framework has been adapted 
from knowledge engineering work in medical 
diagnosis to represent the mental events of 
troubleshooting as conditional ized knowledge 
(Clancey, 1985). In this framework, actions of 
the problem solver are recorded as discrete 
operations or procedures, e.g., tracing 
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schematics or measuring voltage. In addition, 
reasons or precursors for the actions are 
expressed as the goals or intents of the problem 
solver, and the interpretations of outcomes 
resulting from the actions are recorded as 
well. Finally, block diagram-like sketches of 
the equipment parts that are affected by the 
outcomes and actions are generated by the 
technician to illustrate the series of steps. 
Sequences of mental events such as these are 
called PARI structures (precursor [to Action] - 
Action - Result - interpretation). An example 
of PARI data for a single action node is shown 
in Table I. 

Notice in this PARI example that the Action 
element is a familiar troubleshooting procedure, 
namely, taking a voltage measurement with a 
multimeter. The representational formalism of 
the PARI framework does more than reveal that a 
technician needs to know how to take a voltage 
measurement, however. What is also captured are 
the conditions that surround such a measurement 
operation, including the reasons behind the 
action (..."to see if the signal is good up to 
test package cable") and the interpretation of 
an expected voltage level (..."tells me... that 
part of stimulus path [upstream] is good"). In 
effect, the vital strategic processes of 
troubleshooting are made explicit with this 
representation scheme. The plan that produced 
the measurement operation becomes knowfu The 
technician's plan is to constrain the problem 
space by eliminating either the stimulus or 
measurement (return) portion of the signal 
path. It is precisely this kind of strategic 
skill that too often goes "untaught" in 
electronics training, in much the same way that 
strategic knowledge is frequently ignored in the 
teaching of mathematics (Greeno, 1978). When 
problem solving performances are captured in 
real time, it becomes possible to engineer 
strategic knowledge for input to instructional 
systems along with the more standard declarative 
knowledge. In this manner a skill such as 
taking a voltage reading is represented in terms 
of its ties to the conditions of use, just as it 
occurs in real world expert performances. 

Representing skill components in this form 
offers considerable power to instruction, given 
that condi tionali zed knowledge is a recognized 
hallmark of expertise. Conversely, novices 
often display fragmented, unprincipled behavior 
that suggests weakness in the procedural!* zing 
(or conditional i zing) of their skill 
components. In the present example, novices may 
know how to use a multimeter to take a voltage 
reading but often do not produce that action 
under the appropriate conditions. If produced, 
they often have difficulty interpreting the 
results of the action. 

KNOWLEDGE ENGINEERING RESULTS 

Approximately 15 technical experts and 200 
less-than-expert technicians in four related AF 
electronics specialties have participated to 
date in knowledge engineering studies similar to 
those described above as part of the Basic Job 


Skills Research Program. On the basis of these 
studies, a meaningful superstructure for 
organizing troubleshooting performance data has 
been developed. It consists of three 
components, one of which is strategic knowledge 
as previewed above. The three interacting 
components are (1) system knowledge or the 
equipment device models experts use in problem 
solving (e.g., system knowledge regarding the 
stimulus or measurement functionalities of the 
equipment); (2) troubleshooting procedures or 
operations performed on the system; and (3) 
strategic knowledge, which includes (a) 
strategic decision factors that involve fault 
probabilities and efficiency estimates and (b) a 
top-level plan or strategy that is responsible 
for the orchestration of skill components in 
task execution. The orchestration occurs as the 
Strategy component, which sits on top of the 
Procedures and System Knowledge components, 
deploys pieces of knowledge and procedural 
subroutines as needed and as driven by the 
decision factors (Figure 1). 

The System Knowledge component of the 
architecture deserves special attention for 
several reasons. First, it provides the 
dominant organizing principle for this cognitive 
skills architecture. It is the foundation to 
which the companion Procedures component in 
Figure 1 is attached. According to this view, a 
measurement or swapping operation is attached to 
a device model representation, since the purpose 
of the operation is viewed as adjusting the 
technician's present model of the device with 
new knowledge of faulty components. This 
"attachment" is part of the conditional ized 
character of expert knowledge. System Knowledge 
also feeds the strategic decision factors that 
underlie the Strategy component, since these 
factors involve system fault probabilities and 
efficiency estimates associated with operations 
on the system, e.g., it is judged time efficient 
by experts to run self diagnostics on some 
pieces of equipment but not others. Finally, 
System Knowledge influences the goal structure 
of the Strategy component in the sense that 
certain areas of the equipment are targeted 
before others (again due to fault probabilities 
and efficiency considerations). 

The second reason why System Knowledge merits 
special attention here is because the curriculum 
content for the intelligent tutor described in 
the next section is directly influenced by the 
different system perspectives of expert 
troubleshooters. In the course of the knowledge 
engineering studies conducted to date in the BJS 
project, it has become clear that experts' 
decision making during troubleshooting is 
partially driven by system schemas. The schemas 
represent a set of system-related questions that 
experts entertain at various stages in the fault 
isolation process (Collins, 1987). They include 
the following: 

-Is the system fail a glitch, an intermittent 
fail, or a hard fail? 

-In which large functional area of the 
equipment--i.e. , Line Replaceable Unit (LRU), 
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Test Package, or Test Station— is the fault 
located? 

-Is the problem a power-related fail? 

-Is the problem a stimulus or measurement 
problem? 

-Is the problem a signal or data flow problem? 
-Do the symptoms indicate the fault is in a 
device or in the connections between devices? 

These questions can be viewed as the major 
parses the expert makes of the fault isolation 
space in which he/she works. Three of these 
parses have provided the framework for the 
troubleshooting problems that comprise the 
instructional content for the avionics 
intelligent tutor to be described next, 

A SIMULATED MAINTENANCE WORK ENVIRONMENT 

An intelligent maintenance practice environment 
for FI 5 integrated avionics technicians has been 
developed by researchers at the University of 
Pittsburgh’s Learning R&D Center in 
collaboration with AF technical experts 
(Lesgold, 1987). The tutor is based on results 
from cognitive analyses of expert and novice AF 
technicians using the knowledge engineering 
methods referenced above. The analyses have 
provided three general types of input to the 
intelligent tutoring system: detailed 

characterizations of expert performance which 
are the targets for instruction (expressed in 
terns of the cognitive skills architecture of 
Figure 1); a framework for the design of the 
troubleshooting curriculum based on three parses 
experts make of the problem space in this 
domain; and guidelines for the instructional 
treatment based on expert-novice differences as 
well as on present impediments to apprenticeship 
learning in the workplace. 

Expert Parses 

Two central system schemas that experts activate 
as they navigate and parse problem spaces in 
this domain have provided the design framework 
for the maintenance tutor's problem set. These 
schemas represent two system perspectives 
experts' invoke, depending upon the conditions 
of the problem. The first concerns the major 
functionalities of the equipment, namely, 
stimulus and measurement functions. Recall that 
in the example reported in Table I the expert 
both explains his action and interprets the 
system's response to the action in terms of the 
stimulus portion of the equipment. More 
specifically, the procedure (action) used allows 
him/her to achieve the goal of verifying that a 
major functional area of the equipment is 
operating properly. 

The stimulus -measurement functionalities of this 
equipment are illustrated in Figure 2, This is 
an abstracted characterization of the system's 
signal path. As shown, the signal originates in 
the stimulus drawer of an avionics test station, 
travels through the station's switching drawer 
(S/C) which performs signal switching functions, 
and through an interface test package to an 
aircraft line replaceable unit (LRU) which is 


being tested for a malfunction. It returns 
through the interface package to a measurement 
source in the test station. 

Problems in the tutor curriculum represent 
faults at varying levels of difficulty in the 
stimulus and measurement routing of the 
equipment. Trainees will have modeled for them 
how an expert uses this perspective to isolate 
various faults. They will then have extensive 
opportunities to solve problems— with the 
assistance of a hint -giving coach— so that 
system functionality knowledge is tied to 
problem solving conditions. This kind of 
learning environment is in contrast to 
instruction where system knowledge would be 
taught as declarative facts detached from the 
conditions of use, or where measurement 
procedures would be taught in isolation from the 
system and the fault isolation context. 

Results of our knowledge engineering work plus 
input from the dominant theory of skill 
acquisition in psychology today (Anderson, 1982) 
have shaped this instructional approach. First, 
our results have indicated that a principal form 
of the conditionalized knowledge of experts in 
this domain is the coupling of conceptual system 
knowledge (e.g., the stimulus -measurement 
functionality) with procedural and strategic 
components. This results in experts' 
investigating their equipment with specific 
intents and particularized procedures. In other 
words their system knowledge is not detached and 
inert, but rather is tightly interwoven with 
problem solving actions that are produced by 
strategic plans in response to certain 
malfunction conditions. Presently, in the Air 
Force this form of conditionalized knowledge 
results only after many years of experience, as 
would be predicted by Anderson's theory. A 
principal goal of the BJ$ maintenance tutor is 
to speed up that condi tionali zing process. 

The second system perspective or schema used to 
shape the tutor's problem set is signal flow vs 
data flow. Experts also view the equipment (and 
thus represent faults) in terms of these two 
interrelated system properties. In short, this 
schema involves knowledge that both an 
electronic signal and instructions (control 
data) to the equipment for handling the signal 
move through the system. Faults can occur with 
respect to either property. Accordingly, signal 
flow and control data flow problems are 
incorporated in the tutor at varying levels of 
difficulty. 

Finally, a third schema, namely, the macro level 
functional representation of the equipment (LRU 
vs Test Package vs Test Station) has guided 
problem development. This schema is integrally 
tied to experts' strategic planning knowledge in 
the sense that they typically plan their moves 
through the problem space so that they system- 
atically rule out the LRU before moving their 
focus to either the Test Package or Test Sta- 
tion. Trainees will make decisions within the 
tutor environment to pursue either an LRU Plan, 
a Test Package Plan, or a Test Station Plan. 
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In summary, the development of the Air Force 
avionics tutor illustrates that knowledge 
engineering can usefully feed instructional 
design as well as provide the more standard type 
of input, i.e., the expert knowledge base. 
Further, dynamic, problem-based knowledge 
engineering allows for the representation of 
condi tionali zed knowledge so that the most 
critical stage of skill acquisition can be 
targeted by instruction. That is the stage at 
which knowledge becomes tied to conditions of 
use. The avionics maintenance tutoring system 
based on this approach will be discussed in more 
detail in the next section. 

An AI Instructional Application. 

The BJS tutoring system that has resulted from 
the expert dyad approach to knowledge 
engineering is an interesting AI application in 
the sense that it embodies minimally deep 
intelligence. It avoids complete qualitative 
physics of the work envi ronment as well as a 
complete computer representation of expertise 
(Lesgold, 1987). In short, there is neither a 
fully articulate expert nor a runnable equipment 
simulation. Later tutors in the BJS series will 
have these features; however, this initial 
system is of special interest in its own right. 
Its development is much less resource intensive 
than that of deep intelligence tutors, and it 
has received an enthusiastic reception from 
technical experts at the three operational sites 
where it will soon be tested. If the evaluation 
results reveal troubleshooting performance gains 
in accordance with the predictions of field 
personnel, this form of intelligent tutoring 
system represents a quite feasible prototype 
that can immediately generalize to other 
troubleshooting domains. 

A rigorous evaluation study will accompany the 
intervention in order to formally assess its 
effectiveness. A controlled experiment will 
permit the determination of how much on-the-job 
experience is replaced by the 30 to 50 hours of 
tutor Instruction. In addition, performance of 
individual technicians and the shop-level 
productivity of the three FI 5 workcenters will 
be tracked longitudinally to ascertain the 
long-term impact of the instruction. 

As a precursor to this series of BJS intelligent 
tutoring systems, a training study involving a 
human tutor (versus a computer coach) was 
conducted in a related FI 5 integrated avionics 
domain. One goal was to test the concept of 
basing instruction on representations of 
condi tionali zed expert knowledge. The treatment 
involved the posing of authentic troubleshooting 
problems similar to those generated in a BJS 
knowledge engineering study as described above. 
The expert-like skills targeted for enhancement 
were particular instantiations of the cognitive 
skills architecture (Figure 1). The system 
knowledge of interest was the abstracted signal 
path shown in Figure 2, plus several layers of 
elaborated system knowledge. The procedures of 
interest were three methods for investigating 
the equipment that ranged from rudimentary to 


advanced: 

(1 ) swapping equipment components 

(2) using self-diagnostics to test system 
integrity 

(3) measuring device and circuit functionality. 

Increasingly complex system and strategic 
knowledge are associated with increasingly 
sophisticated methods. 

During three to five hours of individual 
instruction over a period of three days, seven 
technicians were tutored. They were presented a 
troubleshooting scenario and then probed 
regarding what they would do to isolate the 
fault (Actions), why they would take the 
particular action (Precursors), and what the 
outcome (Result) of the action meant to them 
(Interpretation). In effect, technicians were 
instructed to generate PARI records (see Table 
1) including the associated device model 
sketches. The human tutor gave feedback to 
their stated Precursors, Actions, and 
Interpretations in the form of hints intended to 
move them toward more expert-like performances. 

To evaluate their learning, they were given both 
an end-of-training problem-based test as well as 
a delayed posttest after the weekend. The tests 
were authentic troubleshooting scenarios 
belonging to the same class and difficulty of 
problems on which they had been tutored. Their 
progress was scored both in terms of the 
sufficiency of their operations— that is, 
whether they sufficiently investigated all 
suspect pieces of the equipment— and the 
efficiency of their moves— that is, whether they 
efficiently conserved time and equipment 
resources. 

Results showed statistically significant 
improvements in both areas, with particularly 
dramatic gains in efficiency. Mean scores are 
plotted in Figure 3. The group's Sufficiency in 
examining all suspect parts of the equipment 
improved from a pretest mean value of 84 
(range = 60 to 95) to a posttest mean of 100. 

The dalayed posttest mean was also 100, 
indicating the improvement was retained over the 
weekend. The group's Efficiency in fault 
isolation improved over twofold. The mean 
pretest value was 37 (range = 24 to 52); the 
initial posttest mean was 92 (range = 81 to 
100); and the delayed posttest mean was 93 
(range =81 to 97). 

Pedagogical ly, this human tutor training study 
was based on the same instructional principles 
that underpin the computer-based avionics 
tutor. Technicians were afforded extensive 
practice in fault-isolation; they were required 
to articulate and focus on their reasons and 
their interpretations of various troubleshooting 
moves; they were aided by a human tutor who, 
principally through Socratic dialogue, 
challenged them to reflect on what they did in 
terms of expert standards of thoroughness and 
efficiency. The technicians later attributed 
the gains they made to the opportunities they 
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had to practice fault Isolation procedures 
intensively and to solve problems 
independently. They reported that recording and 
reflecting on their actions and reasons was 
helpful and that they profitted from the hints 
and consistent feedback. This successful study 
is viewed as empirical support for the 
effectiveness of skill acquisition treatments 
that focus on the conditional i zing of knowledge 
in intelligent learning environments. External 
support in the form of the PARI records and the 
human tutor's feedback appeared to play a 
central role in learning. Finally, the 
instruction was realizable because of the 
knowledge engineering input that revealed the 
processes by which experts condi tionalize what 
they know. 
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Table 1: PARI DATA 


Precursor: Want to see if the stimulus signal is good up to test package cable 

Action: Measure signal at J14-28 with multimeter 
Result: 28 volts 

Interpretation: This Is expected reading; this tells me that the stimulus 

is getting from the test station through the cable, so that 
part of the stimulus path is good 
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